Man-machine interface unit control method, robot apparatus, and its action control method

ABSTRACT

There is proposed a method that may be universally used for controlling a man-machine interface unit. A learning sample is used in order at least to derive and/or initialize a target action (t) to be carried out and to lead the user from an optional current status (e c ) to an optional desired target status (e t ) as the final status (e f ). This learning sample (l) is formed by a data triple made up by an initial status (e i ) before an optional action (a) carried out by the user, a final status (e f ) after the action taken place (a).

TECHNICAL FIELD

This invention relates to a method for controlling a man-machineinterfacing unit, a robot apparatus, and to a method for controlling thebehavior. More particularly, it relates to a method for controlling aman-machine interfacing unit having a step of learning the performanceand/or the entertainment capability, a robot apparatus to which theman-machine interfacing unit controlling method is applied, and to amethod for controlling the behavior.

BACKGROUND ART

Nowadays, a large variety of equipment and appliances employman-machine-interface techniques, man-machine-dialogue systems, and/orthe like to ensure an easy and reliable use of the equipment and toincrease the user's convenience.

Prior art method and systems for controlling man-machine-interface unitsinvolved in such equipment and appliances use predetermined rules whichare derived from common knowledge and/or from user studies to generateappropriate actions and behavior of a machine with respect to thebehavior and the expectations of the user.

These given predetermined rules are essentially fixed and theperformance capability of such a method and system for controlling aman-machine-interface unit is limited to a set of generic and/oruser-independent rules and actions. Although this might be the bestsolution on average, it is in most cases, i.e. for most specific users,not an optimal solution.

DISCLOSURE OF THE INVENTION

It is an object of the present invention to provide a method forcontrolling a man-machine-interface unit with a flexible and reliableresponse to the behavior, expectations and inquiries of a human user.

The object is achieved by a method for controlling aman-machine-interface with the features of claim 1. Preferredembodiments of the inventive method for controlling aman-machine-interface unit are within the scope of the dependentsub-claims. Additionally, the object is achieved by a system forcontrolling a man-machine-interface according to the features of claim19 and by a computer program product according to the features of claim20.

In the inventive method for controlling a man-machine-interface unitwith respect to at least one user an initial status before and a finalstatus after an action taken place are detected. Further, data triplesessentially representing the initial status, the final status, and theaction taken place are collected as learning samples. The learningsamples are used to at least derive and/or initialize a target action tobe carried out so as to carry in each case the user from a given currentstatus to a given desired target status.

It is therefore a basic idea of the present invention to derive for atleast one user with respect to an action which is carried out—inparticular by the man-machine-interface unit itself and/or by anexecution unit connected thereto—an initial status of the user beforethe action has taken place and a final status immediately after theaction has taken place or of data corresponding to or beingrepresentative for the initial status and/or the final status of theuser. Then, learning samples are generated as data triples or tripletswhich essentially represent the initial status, the final status and theaction which has been carried out between the initial status and thefinal status. Furtheron, the learning samples are collected and used toat least derive and/or initialize target actions to be carried out totransform a given user from a given current status to a given desiredtarget status by means of the target action.

In contrast to prior art method for controlling a man-machine-interfaceunit the inventive method is particular flexible, as the method does notfocus on a given, fixed and predetermined set of rules which areconsequently user-independent. In contrast, the inventive methodalternatively or additionally obtains data from the situation takingplace and the interaction between the user and the man-machine-interfaceunit. Data are measured and obtained which correspond to the status ofthe user before and after a performed action. Therefore, the actioncarried out connects the initial status of the user before the actionwith the final status of the user after the action.

It is preferred for representing the different statuses in each case toreceive and/or generate current or initial status data. These current orinitial status data essentially correspond to and/or represent thecurrent status and/or the initial status of the user, respectively.

Furtheron, in each case final or target status data are received and/orgenerated. These final or target status data essentially correspond toand/or represent the final status and the target status of the user,respectively.

Additionally, action data and target action data may be received and/orgenerated essentially corresponding to and/or representing the actiontaken place and the target action derived in accordance with a furtherpreferred embodiment of the inventive method.

A natural and canonical description and representation of the differentstatuses and statuses is achieved in accordance with a preferred andadvantageous embodiment of the present invention according to which thecurrent, initial, final and/or target statuses and in particular therepresenting and/or corresponding data thereof are in each caserepresented by and/or modeled in an embedding topological manifold, inparticular of two dimensions.

Alternatively or additionally the action taken place, the target actionand/or the representing and/or corresponding data thereof may be in eachcase represented by and/or modeled in an embedding topological manifold.

Of course, the above-mentioned topological manifolds may be built up,derived and/or used as separate manifolds for each distinct user.Alternatively or additionally, a plurality of users may be subdividedinto classes of equivalent users and for each of the classes may bebuilt up, derived and/or used a separate manifold. Alternatively, commonor global manifolds for all users may be derived and/or used. This mayalso be done as an additional measure.

Although several properties of the users may be involved to describe thedifferent statuses, it is preferred in accordance with anotheradvantageous embodiment of the present invention to use emotionalstatuses or the like as the current, initial, final, target statuses ofthe user.

Each action taken place or to be performed can be understood as anentity which connects at least one initial status with at least onefinal or target status of the user. Therefore, pairs of initial statusesand certain actions are connected in some sense with a final status. Asall statuses and actions may be understood as topological manifolds amapping—called transmotion mapping T—may be defined for each user basedon the set of all learning samples. The transmotion mapping T may begiven by the following equation (1):T:E⊕×A→E <ei.a>ε⊕A→ef:=T(ei,a)=Ta(ei)εE  (1)wherein E denotes an emotion manifold and is one of the topologicalmanifolds for the user statuses or for the status data thereof, Adenotes an action manifold and is one of the topological manifolds forthe actions or for the action data and ei, ef, a denote respectiveelements of the manifold E and A, respectively.

To each pair <e, a>of the Cartesian product E×A or the Tensor product E⊕A is assigned another element T(e, a) of the emotion manifold E.Therefore, by the transmotion mapping T and the action manifold A afamily of mappings Ta from E into E is defined.

Of course, the transmotion mapping T is in general user-dependent butmay contain user-independent and/or generic parts or components.

There are many different ways to choose a particular target action totransform a user from a current status to a desired final status. Inaccordance with a preferred embodiment of the inventive method for givendesired initial and/or target statuses for a given user an action from alearning sample with given current status, final status and action takenplace already sampled is chosen as a target action for which the initialand current statuses and/or the final and target statuses and/or thedata thereof are in a next topological neighborhood.

This measure is of particular advantage in the case that at least one ofthe statuses and/or the action do not match exactly. Based on the notionof topological vicinity or neighborhood adjacent samples may be used asequivalent samples with respect to a given learning sample.

It is of particular advantage to use a distance measure, a metric,continuity, direction, vector properties, and/or the like of thestatuses and/or actions with respect to each other to derive and/ordescribe the neighborhoods and/or vicinities.

In some cases it might be possible that desired initial or currentstatuses and final or target statuses of a given user cannot betransformed into each other or connected with another directly. It istherefore advantageous in accordance with a preferred embodiment of theinventive method to derive or initialize a sequence of actions to beexecuted as the target action so as to lead a given user to a targetstatus to fit best to a given desired final status for the user.Thereby, a path through the emotion manifold E is characterized anddefined by the sequence of actions.

In other cases using a given and known action—e.g. from a learningsample already obtained—as a target action might not be sufficient forobtaining a good result when trying to transform a user's initial statusto a final status. Therefore, a modification may be included into and/orperformed on a derived target action and/or the data thereof to achievea good transformation result.

These modification may be introduced randomly by means of aprobabilistic measure, on the basis of topological, distance, metric,continuity, direction, vector properties, and/or the like.

As learning samples can often be obtained only at a slow pace or lowrate—perhaps with a few learning samples every day—it is of particularadvantage to generalize the obtained learning samples in particular toderive the topological manifolds. Therefore, the obtained learningsamples serve as reference points and/or measuring points in thetopological manifold E and A and their generalizations serve asinterpolations between the measuring points.

The step of generalization may comprise a step of averaging and/orweighting learning samples from different users, in particular to derivecommon and/or global topological manifolds. The different users may inparticular be emotionally clustered depending on their typical statusand/or behavior.

Additionally or alternatively, the step of generalizing the learningsamples may at least comprise a step of reducing the dimensions of theaction manifold A, in particular by means of a projection process, orthe like. Thereby, the degrees of freedom are reduced and severalclasses of actions are identified and made equivalent with respect toeach other.

Furtheron, the step of generalizing the learning samples may include astep of enlarging and/or reducing the extension of a neighborhood orvicinity of a given learning sample within which all triples or tripletsof initial statuses, final statuses and target actions are classified asbeing equivalent to the given learning sample. This is in particularperformed until the point where a target action and/or a final statusare valid for all initial statuses. Additionally or alternatively, thismeasure is carried out until the point where the final status isirrelevant and only a direction of a change of the status is considered.

It is a further aspect of the present invention to provide a system, anapparatus, a device, and/or the like for controlling aman-machine-interface unit which is in each case capable of performingand/or realizing the inventive method for controlling aman-machine-interface unit and/or of its steps.

Additionally, it is a further aspect of the present invention to providea computer program product comprising computer program means which isadapted to perform and/or realize the inventive method for controlling aman-machine-interface unit or of its steps when it is executed on acomputer, a digital processing means, and/or the like.

The above-mentioned and further aspects of the present invention willbecome more elucidated taking into account the following remarks:

The performance of a man-machine-interface and/or the goal ofentertaining and responding a user may be achieved by an artificialsystem like an entertainment robot, or the like. In particular, thisgoal can be achieved by estimating a topological map or manifold of useremotions or emotion changes. In a particular simple example thistopological map or manifold may be built up by a 2-dimensional emotionalplane. The topological map or manifold is defined over themultidimensional space of possible or potential system actions. Givensuch a map or manifold the system or method can lead the user to anydesired emotion starting at any given initial emotion. The map functionmay be initialized with some heuristic relationships or with any priorart known up to now. While the method and system is being used itautomatically improves its performance by measuring the users emotions.Thereby, the method and system improves the granularity of the mapand/or creates user-dependent maps or manifolds.

Man-machine-interface units or devices and in particular entertainmentrobots known in the art use predetermined rules which are derived fromuser studies and/or common knowledge. These predetermined rules arefixed and they are used to decide on the type of action to take givingthe goal—for instance to entertain—and potentially the current emotionof the user. However, the predetermined and fixed rules are static andthey are not user-dependent. Some of the proposed prior art methods tofacilitate this decision process can possibly incorporate learning ofuser-dependent strategies to entertain the user and to behave in anappropriate way. However, no good algorithm has yet been proposed thatcan generalize single observations arbitrarily, thereby enabling thesystem to incrementally improve the performance both of theuser-independent and the user-dependent mode.

The variety of human statuses and in particular of human emotions can bespread out as a topological manifold and in particular as a2-dimensional map. In that 2-dimensional map the two axes denotepleasure and arousal. In such a plane the top right area describes moodslike delighted, blissful, excited, or the like statuses. The top leftarea describes for example terrified, angry, disgusted statuses. Thelower right area contains for example content and relaxed mood, whilethe lower left area describes sadness, depression, boredom, and thelike.

Emotions that are close to each other on this manifold or plane aresimilar emotions. It is easier to change the mood from a point on theemotional manifold or plane to another point that is close than tochange to a point in the far distance. This topological property of theemotional manifold or plane is the basis for the invention.

The appropriate action in the task of behaving in appropriate manner orentertaining a user are highly dependent on the current mood and statusof the user and on the desired status or mood change and additionally onthe identity of the user. However, in a man-machine-interface unit andin particular in an entertainment robot product which is to be sold inquantities, the identity of the user is not known beforehand. Hence, theentertainment ability of the system must be limited to a set of generic,user-independent actions. While this is the best solution on average, itis a sub-optimal solution for nearly every specific user. Therefore, theperformance and entertainment ability should be learned by each unit ofan entertainment robot or man-machine-interface unit independently,using its respective owner as an optimization target.

If the system can detect the emotion of its user before and after anyaction has been performed by the system, such a measurement can serve asa learning sample from which future decisions in similar situations canbe derived. However, the learning samples come at a slow pace or lowrate of only a few of them every day. There is so far no method known inthe art how an entertainment robot or a man-machine-interface unit canprofit from every single learning sample if learning samples are fed inincrementally with a priori unknown frequency.

Given a set of samples or learning samples where each sample consists of(a) an initial point on the emotional manifold or plane indicating theinitial user emotion (b) some multidimensional vector describing theaction of the device—for example describing the performance of anentertainment robot or a dialogue strategy of a chatter box—and (c) afinal point on the emotional manifold or plane, indicating the finaluser emotion after the action of the device has been perceived. Then,each such learning sample can be put onto the map or manifold and can bedescribed as a vector starting at the initial point of the emotionalmanifold or plane, pointing to the final point and having an attributeof the multidimensional action vector of the action manifold describedunder (b) above. Each sample can be put into the plane independentlyfrom each other sample. Every user has in general its own emotional mapor manifold where only his learning samples are added, described and/orgeneralized.

If such an emotion manifold or map exists or is derived for a given userit is easy and straightforward to find an appropriate action if anyinitial status or mood and any desired target status or mood of the userare given. This is achieved by selecting an appropriate action vectorstarting at least in a neighborhood or vicinity of the pointcorresponding to the initial status or mood and pointing into the rightand correct direction into a vicinity or neighborhood of a pointcorresponding to the given target mood or status within the emotionmanifold or map of the respective user. Then, the action described bythe distinct action vector is carried out and performed so as totransform the user's initial status to the target or final status. It isbeneficial and improves the generalization performance if not an old andgiven action is repeated but if the old action is slightly and/orrandomly modified. This measure is equivalent to repeating successfulstrategies in an evolutionary way that makes room for furtherimprovements of the strategy.

Since the emotional map will usually be sparsely filled with learningsamples or sample vectors generalizing the set of sample vectors isextremely important for the performance level of the method and thealgorithm. This is in particular true since learning samples come inonly quite slowly and with a low rate—typically only a few of them everyday—but the method and the system need to learn starting from the veryfirst learning sample and also may take into account a set of genericand user-independent starting rules.

This generalization can be accomplished using several techniques. Firstof all, emotion manifolds or maps of users can simply be averaged byadding learning samples of sample vector of different users into oneuser-independent sample. Additionally, global or common emotionmanifolds or maps for classes of users or for all users may be derived.

Second, projections of the multidimensional action vectors or actiondata to reduce the action dimensions can be computed, which reduces thedimensionality of the action manifold or space.

Third, the area in which sample vector or learning sample is consideredto be valid for a given starting emotion can be made larger up to thepoint where a single example is to be considered to be typical for allinitial emotions.

Fourth, the target area can be made larger in a similar way up to thepoint where the target area is regarded as being irrelevant completelyand only the direction of the emotional change vector is considered.

This generalizations can be used independently from each other.Alternatively, they can be combined. Particularly, if only few learningsamples are available, the system can always find a suitable action byincreasing the degree of generalization until a sample action can befound. Thereby, the system will profit even from one single observationsor learning sample and every additional learning sample will increasethe system performance in a very natural and incremental way by allowingit to deduce next actions by generalizing—relaxing constraints—less andless as the emotional map is filled more and more with useful learningsamples.

The new method and algorithm provides incremental improvement of thesystem performance, starting with the very first sample observation,while still allowing a very fine model of emotional changes. The samealgorithm can be used for both cores and fine models.

The above-mentioned and further features and advantages of the inventionwill become more apparent from the following description taking intoaccount the accompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic block diagram showing a preferred embodiment ofthe inventive method for controlling a man-machine-interface unit.

FIG. 2 is a diagram which shows the connection of the differenttopological manifolds and the mapping between them.

FIG. 3 is a perspective view showing the appearance of a robot apparatusembodying the present invention.

FIG. 4 is a block diagram showing a circuit structure of the robotapparatus.

FIG. 5 is a block diagram showing a software structure of the robotapparatus.

FIG. 6 is a block diagram showing the structure of a middleware layer ofthe software structure of the robot apparatus.

FIG. 7 is a block diagram showing the structure of an application layerof the software structure of the robot apparatus.

FIG. 8 is a block diagram showing the structure of a behavior modellibrary of the application layer.

FIG. 9 shows a finite probability automaton which serves as theinformation for determining the behavior of the robot apparatus.

FIG. 10 shows a status transition table provided for each node of thefinite probability automaton.

FIG. 11 shows essential portions of the robot apparatus to which hasbeen applied the method for controlling the man-machine interface unit.

FIG. 12 illustrates the internal structure of a user speech recognitionunit of the robot apparatus.

FIG. 13 illustrates an actuation unit (AU) used for recognition ofexpressions of the user's face.

FIG. 14 illustrates the inner structure of the user image recognitionunit of the robot apparatus.

FIG. 15 illustrates a 3-dimensional topological space for expressing theemotion of the robot apparatus.

FIG. 16 illustrates an instance in which the emotional status is mappedon a point in the 3-dimensional topological space.

FIG. 17 illustrates the manner in which the initial status on atopological plane is transformed into a target status.

FIG. 18 is a flowchart for illustrating the method for controlling thebehavior of the robot apparatus.

BEST MODE FOR CARRYING OUT THE INVENTION

Referring to the drawings, specified embodiments of the presentinvention are explained in detail. The following embodiment is directedto a method and a system for controlling a man-machine interface unit inassociation with the user's operations, expectations or emotion, as abasic principle of the present invention, and to a specified example ofapplication of the present method and system to an autonomous typeentertainment robot having at least the emotion model and which is ableto change the behavior it expresses responsive to the emotion.

(1) Method for Controlling the Man-Machine Interface Unit

Preferred embodiments of the method and the system of the presentinvention for controlling the man-machine interface unit are shown inthe flowchart of FIG. 1, in which the embodiments are illustrated bysteps from S0 to S12.

In a first step S0, the method and the system are initialized withrespect to the onset of the performance of the method. In a followingstep S1, general information and/or a general and/or generic behaviormodel is read and recalled from a predefined data base DB. On the basisof this primary information obtained from the data base DB, the initialoperation of the method and system is constructed.

In the following step S2, an initial status e_(i) or mood or emotion ofa given user is detected as a current status e_(c): e_(i):=e_(c). Thedetected initial status e_(i) is internally represented by certaininitial status data ISD.

In the following step S3, a desired target status to which the usershould be turned to and/or respective target status data TSD arereceived, obtained and/or generated. Distinct data samples can beobtained from the data base DB or they can be obtained from the userdirectly or from an external control unit.

In the next step S4, it is checked whether an appropriate learningsample l=<e_(i)′, a, e_(t)′> approximating in one sense or another agiven data triple <e_(i), a, e_(t)> is available, i.e. it is checked onwhether or not l=<e_(i)′, a, e_(t)′> exits in the data base DB with:l<e_(i), a, e_(t)>.

If this is the case, the learning sample l is received from the database DB in step S5 and then in a following step S6 the action a of thelearning sample l is extracted and defined as the target action t to bederived: t:=a. If an appropriate learning sample l=<e_(i)′, a, e_(t)′>approximating the given data triple <e_(i), a, e_(t)> cannot be found—inparticular in the very beginning of the performance of the method orsystem—a in one sense or another closest sample or learning sample l orgeneric rules are obtained in step S7 from the given data base DB.

In step S8 an appropriate target action t is derived and generated fromthe given information and from a process of generalizing a given genericinformation and/or from the closest learning sample.

After step S6 and/or step S8, i.e. before step S9, the target action maybe modified according to some modification rule and/or randomly.

In a next step S9 the performance of the derived target action t is setand initialized.

Then in a following step S10 the final status e_(f) of the user as aresult of the target action t taken place is detected and in the stepS11 the data triple <e_(i), t, e_(t)> is recalled and evaluated as a newlearning sample l.

In the next step S12, the obtained final status e_(f) of the user iscompared with the desired target status e_(t). Evaluation and comparisondata are obtained and recorded and stored together with a new learningsample <e_(i), t, e_(t)> in the given data base DB. The method thenreturns to the repetition of step S2 and the following steps until theexternal control or the user terminates the prosecution of the methodand the system.

FIG. 2 is a schematic diagram showing the interconnection between theemotion manifolds EM, emotion data manifolds EDM, action manifolds AM,and action data manifolds ADM.

In the upper part of FIG. 2 the Cartesian or tensor product E⊕A—which isconstituted by an emotion manifold E of an user and an action manifoldAM characterizing the man-machine-interface unit or the like—isconnected with the emotion manifold or map via a so-called transmotionmapping T which is in general specific for a given user. Each pair<e_(i), a>εE⊕A—where e_(i) is a given initial status or emotion of theuser and a is an potential action of the system—is mapped via thetransmotion mapping T onto at least one final status or emotion e_(f) Eof the user with e_(f):=T(e_(i), a). Due to the definitionTa(e_(i)):=T(e_(i), a), the action manifold generates a family Ta ofmappings from E into E.

Via a projection operator or process P one arrives at the lower orinternal representation part for the emotion an action manifolds EM anAM, namely to the corresponding emotion data manifold or map EDM and thecorresponding action data manifold or map ADM, which are in this examplerepresented by Cartesian coordinate systems.

The initial status data ISD are therefore generated from the initialstatus e_(i) by means of a measuring, detecting, and representationgenerating process, i.e. the projection process P, with ISD:=P(e_(i)).Additionally, appropriate transmotion data mappings PT and PT_(a) aredefined. Here a final status e_(f) is projected to the emotion datamanifold by FSD:=P(e_(f)). The final status data FSD are connected tothe initial status data ISD via: FSD:=PTa(ISD).

To increase the performance of the inventive method for multiple userappliances an user identification process may be involved. The useridentification may be based on a speech recognition and/or auditoryemotion detection system or process.

(2) Application to Robot Apparatus

The above-described method and system for controlling the man-machineinterface unit finds application in an autonomous type robot apparatushaving at least an emotional model and which is capable of changing theaction to be executed depending on the feeling.

One of the most important and fundamental functions required of therobot apparatus, such as this entertainment robot, is the understandingof the feeling of the human being (Picard R. (1997) Affective Computing,MIT Press). In particular, it is required of the robot apparatus torecognize the feeling of the human being and also to express the robot'sown feeling.

It is noted that, if the robot apparatus is able to change its actionresponsive to the feeling of the human being as user, it enhances theamicability with the human being extremely effectively. Moreover, if, bythe application of the above-described man-machine interface unit, notonly sociability may be enhanced, but also the feeling of the humanbeing may be led to the feeling desired by the robot apparatus, it ispossible to enlarge the extent of the interaction of the robot apparatuswith the human being from the side robot apparatus. This function ismanifested particularly effectively in a robot apparatus having thelearning function.

Thus, the structure of an autonomous robot apparatus, having the feelingmodel and which is capable of changing the action it expresses dependingon the feeling, is first explained, and essential portions of the robotapparatus to which are applied the method and system for controlling theman-machine interface unit are subsequently explained. Although thefollowing explanation is directed to a so-called pet type robot of thetype walking on four feet, any suitable robot apparatus may be usedwhich operates responsive to the feeling model. The performance means isalso not limited to walking on four legs or even to the leggedperformance system.

(2-1) Structure of Robot Apparatus

As shown in FIG. 3, the robot apparatus is a so-called pet robot,simulating an animal, such as a ‘dog’, and is constructed by leg units3A, 3B, 3C and 3D, connected to the front and rear sides of a trunk unit2, and by a head unit 4 and a tail unit 5, connected to the front andrear ends of the trunk unit 2, respectively.

Referring to FIG. 4, the trunk unit 2 includes a controller unit 16,comprised of an interconnection over an internal bus 15 of a CPU(central processing unit) 10, a DRAM (dynamic random access memory) 11,a flash ROM (read-only memory) 12, a PC (personal computer) cardinterface circuit 13 and a signal processing circuit 14, and a battery17 as a power supply for the robot apparatus 1. In the trunk unit 2 arealso housed an angular velocity sensor 18 and an acceleration sensor 19for detecting the orientation and the acceleration of performance of therobot apparatus 1.

On the head unit 4, there are mounted, in position, a CCD (chargecoupled device) camera 20 for imaging an outside status, a touch sensor21, for detecting the pressure resulting from a physical action, such as‘stroking’ or ‘patting’ from the user, a distance sensor 22 formeasuring the distance to an object, positioned ahead, a microphone 23for collecting the external sound, a loudspeaker 24 for outputting thesound, like whining, and LEDs (light emitting diodes) equivalent to the‘eyes’ of the robot apparatus 1.

The joint portions of the leg units 3A to 3D, connecting portions of theleg units 3A to 3D and the trunk unit 2, connecting portions of the headunit 4 and the trunk unit 2 and the connecting portion of a tail 5A ofthe tail unit 5, are provided with a number of actuators 25 ₁ to 25 _(n)and with potentiometers 26 ₁ to 26 _(n) corresponding to the number ofthe degrees of freedom. For example, the actuators 25 ₁ to 25 _(n)include servo motors as constituent elements. The leg units 3A to 3D arecontrolled by the driving of the servo motors to transfer to a targetedorientation or performance.

The sensors, such as the angular velocity sensor 18, acceleration sensor19, touch sensor 21, distance sensor 22, microphone 23, loudspeaker 24and the potentiometers 26 ₁ to 26 _(n), and LED and the actuators 25 ₁to 25 _(n), are connected via associated hubs 27 ₁ to 27 _(n) to thesignal processing circuit 14 of the controller 16, while the CCD camera20 and the battery 17 are connected directly to the signal processingcircuit 14.

The signal processing circuit 14 sequentially captures sensor data,picture data or speech data, furnished from the above-mentionedrespective sensors, to cause the data to be sequentially stored overinternal bus 15 in preset locations in the DRAM 11. In addition, thesignal processing circuit 14 sequentially takes residual batterycapacity data indicating the residual battery capacity supplied from thebattery 17 to store the data thus taken in preset locations in the DRAM11.

The respective sensor data, picture data, speech data and the residualbattery capacity data, thus stored in the DRAM 11, are subsequentlyutilized when the CPU 10 performs actional control of the robotapparatus 1.

In actuality, in an initial stage of power up of the robot apparatus 1,the CPU 10 reads out a memory card 28 loaded in a PC card slot, notshown, of the trunk unit 2, or a control program stored in the flash ROM12, either directly or through a PC card interface circuit 13, forstorage in the DRAM 11.

The CPU 10 then checks its own status and surrounding statuses, and thepossible presence of commands or actions from the user, based on thesensor data, picture data, speech data or residual battery capacitydata, sequentially stored from the signal processing circuit 14 to theDRAM 11.

The CPU 10 also determines the next ensuing actions, based on theverified results and on the control program stored in the DRAM 11, whiledriving the actuators 25 ₁ to 25 _(n), as necessary, based on the sodetermined results, to produce actions, such as swinging the head unit 4in the up-and-down direction or in the left-and-right direction, ormoving the leg units 3A to 3D for walking or jumping.

The CPU 10 generates speech data as necessary and sends the so generateddata through the signal processing circuit 14 as speech signals to theloudspeaker 24 to output the speech derived from the speech signals tooutside or turns on/off or flicker the LEDs.

In this manner, the present robot apparatus 1 is able to behaveautonomously responsive to its own status and to surrounding statuses,or to commands or actions from the user.

(2-2) Software Structure of Control Program

FIG. 5 shows the software structure of the above-mentioned controlprogram in the robot apparatus 1. In FIG. 5, a device driver layer 30 ispositioned in the lowermost layer of the control program, and is formedas a device driver set 31 made up by plural device drivers. Each devicedriver is an object allowed to directly access the hardware used in aroutine computer, such as a CCD camera 20 (FIG. 4) or a timer, andperforms the processing responsive to interruption from an associatedhardware.

A robotics server object 32 is made up by a virtual robot 33, a powermanager 34, comprised of a set of software items responsible forswitching between power sources, a device driver manager 35, comprisedof a set of software items, supervising various other device drivers,and a designed robot 36, comprised of a set of software itemssupervising the mechanism of the robot apparatus 1. The virtual robot33, located in the lowermost layer of the device driver layer 30, iscomprised of a set of software items furnishing an interface foraccessing the hardware items, including the above-mentioned varioussensors and actuators 25, to 25 n.

A manager object 37 is made up by an object manager 38 and a servicemanager 39. The object manager 38 is a set of software items supervisingthe booting and the end of respective software items included in therobotics server object 32, a middle ware layer 40 and an applicationlayer 41, while the service manager 39 is a set of software itemssupervising the connection to respective objects based on theinformation on the connection among respective objects stated in aconnection file stored in a memory card 28 (FIG. 4).

The middle ware layer 40 is positioned as an upper layer of the roboticsserver object 32, and is made up by a set of software items providingbasic functions of the robot apparatus 1, such as picture processing orspeech processing. The application layer 41 is located as an upper layerof the middle ware layer 40, and is a set of software items for decidingon the behavior of the robot apparatus 1 based on the results of theprocessing by the software items making up the middle ware layer 40.

FIG. 6 shows specified software structures of the middle ware layer 40and the application layer 41.

Referring to FIG. 6, the middle ware layer 40 is made up by arecognition system 60, having signal processing modules 50 to 58 fornoise-, temperature- or lightness detection, sound scale recognition,distance- or orientation detection, for a touch sensor, for motiondetection and for color recognition, and an input semantics convertermodule 68, and by an outputting system 69, having an output semanticsconverter module 68 and signal processing modules 61 to 67 fororientation management, tracking, motion reproduction, walking,restoration from the falldown status, LED lighting and for soundreproduction.

The signal processing modules 50 to 58 of the recognition system 60takes relevant data from the sensor data, picture data and the speechdata, read out by the virtual robot 33 of the robotics server object 32from the DRAM 11 (FIG. 4), to process the data, and routes the processedresults to the input semantics converter module 59. It is noted that thevirtual robot 33 is constructed as a component for exchanging orconverting signals in accordance with a preset communication protocol.

The input semantics converter module 59 recognizes the own status, thesurrounding status, user's commands or actions, such as ‘annoying’,‘sultry’, ‘light’, ‘a ball has been detected’, ‘falldown is detected’,‘stroked’, ‘patted’, ‘do-mi-so scale has been heard’, ‘a moving objecthas been detected’, or ‘an obstacle has been detected’ to output theresults of recognition to the application layer 41 (FIG. 5).

The application layer 41 is made up by five modules, namely a behaviormodel library 70, a behavior switching module 71, a learning module 72,a feeling model 73 and an instinct model 74, as shown in FIG. 7.

In the behavior model library 70, there are provided respectiveindependent behavior models 70 ₁ to 70 _(n) in association with pluralpre-selected condition items, such as ‘residual battery capacity issmall’, ‘restoration from the faildown status’, ‘an obstacle is to beevaded’, ‘the feeling is to be expressed’ or ‘a ball has been detected’,as shown in FIG. 8.

When the results of recognition are provided from the input semanticsconverter module 59 or a preset time has elapsed as from the time thelast results of recognition were provided, the behavior models 70 ₁ to70 _(n) decide on the next behaviors, as they refer to parameter valuesof the emotion as held by the feeling model 73 and to parameter valuesof the corresponding desires as held by the instinct model 74, to sendthe results of decision to the behavior switching module 71.

In the present embodiment, the behavior models 70 ₁ to 70 _(n) use analgorithm, termed finite probability automaton, as a technique ofdeciding on the next behavior. This algorithm probabilisticallydetermines from which one of the nodes (statuses) NODE₀ to NODE_(n), asshown in FIG. 9, to which one of these nodes NODE₀ to NODE_(n)transition is to be made, based on the values of the transitionprobability P₁ to P_(n) as set for the arcs ARC₁ to ARC_(n)interconnecting the respective nodes NODE₀ to NODE_(n).

Specifically, each of the behavior models 70 ₁ to 70 _(n) includes astatus transition table 80, shown in FIG. 10, for each of the nodesNODE₀ to NODE_(n), forming the own behavior models 70 ₁ to 70 _(n), inassociation with these nodes NODE₀ to NODE_(n).

In the status transition table 80, input events (results ofrecognition), as the conditions for transition in the nodes NODE₀ toNODE_(n), are listed in the column of the ‘input event name’, in thepriority order, and further conditions for the transition condition arestated in associated rows of the columns ‘data name’ and ‘data range’.

Thus, in the node NODE₁₀₀, shown in the status transition table 80 ofFIG. 10, given the results of recognition ‘ball has been detected’(BALL), the ball size (SIZE) being ‘from 0 to 1000’, as given along withthe results of recognition, represents a condition for transition toanother node. Similarly, given the results of recognition ‘an obstaclehas been detected’ (OBSTACLE), the distance (DISTANCE) to the obstaclebeing in a range ‘from 0 to 100’, as given along with the results ofrecognition, represents a condition for transition to another node.

Also, in the present node NODE₁₀₀, if no results of recognition areinput, but any one of the parameter values ‘joy’ (JOY), surprise(SURPRISE) or ‘sadness’ (SADNESS) among the parameter values of therespective emotion and desires, as held in the feeling model 73, amongthe parameter values periodically referenced by the behavior models 70 ₁to 70 _(n), is in a range between'50 and 100′, transition may be made toanother node.

Moreover, in the status transition table 80, the node names to whichtransition can be made from the node NODE₀ to NODE_(n) are shown in therow ‘nodes of destination of transition’ in the column ‘probability oftransition to other nodes’. Additionally, the probability of thetransition to other nodes NODE₀ to NODE_(n), enabled when all conditionsstated in the columns ‘input event name’, ‘data name’ and ‘data range’are met, is entered in corresponding locations in the column‘probability of transition to other nodes’. The behaviors to be outputon the occasion of transition to the nodes NODE₀ to NODE_(n) areindicated in the row ‘output behavior’ in the column ‘probability oftransition to other nodes’. Meanwhile, the sum of the probability valuesof each row in the column ‘probability of transition to other nodes’ is100%.

Thus, in the node NODE₁₀₀ represented by the status transition table 80of FIG. 10, given the results of recognition that ‘the ball has beendetected’ and that the size (SIZE) of the ball is in a range from ‘0 to1000’, transition to the ‘node NODE₁₂₀ (node 120)’ can be made with theprobability of 30%, and the behavior ‘ACTION 1’ is then output.

In each of the behavior models 70 ₁ to 70 _(n), a plural number of thesets of the nodes NODE₀ to NODE_(n), each stated as this statustransition table 80, are concatenated together, such that, given theresults of recognition from the input semantics converter module 59, thenext behavior is stochastically determined by exploiting the statustransition tables of the NODE₀ to NODE_(n) and the results of thedecision are output to the behavior switching module 71.

The behavior switching module 71, shown in FIG. 7, sends to the outputsemantics converter module 68 of the middle ware layer 40 a command toselect the behavior output from one of the behavior models 70 ₁ to 70_(n), having a preset high priority order, among the behaviors outputfrom the respective behavior models 70 ₁ to 70 _(n) of the behaviormodel library 70, and to execute the behavior. This command is referredto below as a behavior command. In the present embodiment, the order ofpriority of a given one of the behavior models 70 ₁ to 70 _(n) shown inFIG. 8 is the higher the lower the rank of the behavior model inquestion in FIG. 8.

The behavior switching module 71 notifies the learning module 72,feeling model 73 and the instinct model 74 of the effect of thetermination of the behavior, based on the behavior completioninformation afforded from the output semantics converter module 68 afterthe end of the behavior.

The learning module 72 is fed with the results of recognition of theinstructions received as an action from a user, such as ‘patting’ or‘stroking’, from among the results of recognition provided from theinput semantics converter module 59.

The learning module 72 changes the probability of transition of thebehavior models 70 ₁ to 70 _(n) in the behavior model library 70, basedon the results of recognition and on the notification from the behaviorswitching module 71, such that, when the action is ‘patting’(‘scolding’) or ‘stroking’ (‘praising’), the probability of occurrenceof the behavior in question will be increased or decreased,respectively.

On the other hand, the feeling model 73 is holding parametersrepresenting the intensity of each of the six emotion types, namely joy(JOY), sadness (SADNESS), anger (ANGER), surprise (SURPRISE), disgust(DISGUST) and fear (FEAR). The feeling model 73 periodically updates theparameter values of these emotion types, based on the particular resultsof recognition provided by the input semantics converter module 59, suchas ‘patted’ or ‘stroked’, time elapsed and on the notification from thebehavior switching module 71.

Specifically, the feeling model 73 calculates a parameter value E[t+1]of the current emotion type for the next period in accordance with thefollowing equation (2):E[t+1]=E[t]+ke×ΔE[t]  (2)where ΔE[t] in the amount of variation of the emotion type as calculatedby a preset equation based on, for example, the results of recognitionprovided by the input semantics converter module 59, the behavior of therobot apparatus 1 at the pertinent time or on the time elapsed as fromthe previous updating event, E[t] is the current parameter value of theemotional type and k_(e) is a coefficient representing the sensitivityof the emotion type. The feeling model 73 substitutes the so calculatedvalue for the current parameter value E[t] of the emotion type to updatethe parameter value of the emotion type. In similar manner, the feelingmodel 73 updates the parameter values of the totality of the emotiontypes.

Which effect the respective results of recognition and the notificationfrom the output semantics converter module 68 will have on the variationof the parameter values of the respective emotion types ΔE[t] is hasbeen set beforehand, in such a manner that the results of recognition‘patted’ significantly affects the amount of variation ΔE[t] of theparameter value of the emotion type ‘anger’, while the results ofrecognition ‘patted’ significantly affects the amount of variation ΔE[t]of the parameter value of the emotion type ‘joy’.

The notification from the output semantics converter module 68 is theso-called behavior feedback information (behavior end information) andthe information concerning the results of occurrence of the behavior.The feeling model 73 also changes the feeling based on this information.For example, the feeling level of anger may be lowered by the act of‘barking’. Meanwhile, the notification from the output semanticsconverter module 68 is also input to the learning module 72, which thenchanges the corresponding transition probability of the behavior models70, to 70 based on this notification.

Meanwhile, the feedback of the results of the behavior may be made by anoutput of the behavior switching module 71 (behavior seasoned with thefeeling).

On the other hand, the instinct model 74 holds the parameters,representing the intensity of five reciprocally independent desires,namely ‘desire for exercise’ ‘desire for affection’, ‘appetite’ and‘curiosity’. The instinct model 74 periodically updates the parametervalues of these desires, based on the results of recognition providedfrom the input semantics converter module 59, time elapsed and on thenotification from the behavior switching module 71.

Specifically, as concerns the ‘desire for exercise’, ‘desire foraffection’ and ‘curiosity’, the instinct model 74 calculates, at apreset period, the parameter value I [k+1] of these desires at the nextperiod, using the following equation (3):I[k+1]=1[k]+ki×ΔI[k]  (3)where ΔI[k] is the amount of variation of the desire in question at apertinent time as calculated by a preset equation based on the resultsof recognition, time elapsed and the notification of the outputsemantics converter module 68, I[k] is the current parameter value ofthe desire and ki is the coefficient representing the sensitivity of thedesire in question, and substitutes the calculated results for thecurrent parameter value I[k] to update the parameter value of thedesire. The instinct model 74 updates the parameter values of therespective desires except the ‘appetite’.

The effect of the results of recognition and the notification from theoutput semantics converter module 68 on the amount of variation ΔI[k] ofthe parameter values of the respective desires is set beforehand, suchthat, for example, the notification from the output semantics convertermodule 68 significantly affects the amount of variation ΔI[k] of theparameter values of ‘fatigue’.

In the present embodiment, the parameters of the respective emotiontypes and the respective desires (instincts) are varied in a range from0 to 100, while the values of the coefficients k_(c) and k_(i) are alsoset individually for the respective emotion types and for respectivedesires.

The output semantics converter module 68 of the middle ware layer 40sends abstract behavior commands, such as ‘go ahead’, ‘joy’, ‘cry’, or‘tracking (track a ball)’, provided by the behavior switching module 71of the application layer 41, as described above, to the signalprocessing modules 61 to 67 of the output system 69, as shown in FIG. 6.

Given a command for a behavior, the signal processing modules 61 to 67generates servo command values to be supplied to the associatedactuators 25 ₁ to 25 _(n)(FIG. 4) to execute the behavior, speech dataof the sound to be output from the loudspeaker 24 (FIG. 4) and/ordriving data to be supplied to the LED of the ‘eye’, based on thebehavior command, and send these data through the virtual robot 33 ofthe robotics server object 32 and the signal processing circuit 14 (FIG.4) in this order to the associated actuators 25 ₁ to 25 _(n),loudspeaker 24 or to the LED.

In this manner, the robot apparatus 1 is able to perform an autonomousbehavior, based on the control program, responsive to its own internalstatus, surrounding status (exterior status) or to the command or actionfrom the user.

(2-3) Relevant Portion in the Robot Apparatus

The essential portions of the robot apparatus 1, to which theabove-described method and system for controlling the man-machineinterface unit are applied, is hereinafter explained.

Apart from the expression with speech, there are two ways of feelingexpression of the human being. These are changes in facial expression(Eckman, P. (1982) Emotions in the human face, Cambridge Universitypress, Cambridge), changes in the voice intonation (Banse, R. andSherer, K. R., (1966) Acoustic Profiles in Vocal Emotion Expression,Journal of Personality and Social Psychology, 70(3): 614-636).

It is assumed that the robot apparatus 1 of the present embodimentestimates the user's feeling based on the picture signals and/or speechsignals obtained from the user to cause the action to be changed basedon the estimated results.

Specifically, the robot apparatus 1, as shown in FIG. 11, includes aninput unit 101, an internal status model 102, a user speech recognitionunit 103, a user image recognition unit 104, a short term memory (STM)105, a feeling estimation unit 106, a long term memory (LTM) 107 and anaction selection unit 108.

To the input unit 101, the sensor information from various sensors, suchas touch sensor 21, is input, in addition to picture signals from theCCD camera 20 or speech signals (acoustic signals) from the microphone23 shown in FIG. 4.

The internal status model 102 corresponds to the feeling model 73 andthe instinct model 74, shown in FIG. 7, and sends emotion parameters,representing any of joy (JOY), sadness (SADNESS), anger (ANGER),surprise (SURPRISE), disgust (DISGUST) and fear (FEAR), and desireparameters, representing any of desires for exercise (EXERCISE),affection (AFFECTION), appetite (APPETITE) and curiosity (CURIOSITY), tothe feeling estimation unit 106.

When supplied with the speech data, corresponding to the speech utteredby the user, from the input unit 101, the user speech recognition unit103 detects the feeling, as later explained, based on the intensity ofthe speech and the fundamental frequency (pitch), to send the sort ofthe emotion as detected and its likelihood to the short term memory 105.

When supplied with the user image data from the input unit 101, the userimage recognition unit 104 extracts the face region from the picture, aslater explained, to send the sort of the emotion as detected and itslikelihood to the short term memory 105.

The short term memory 105, responsible for short-term storage for therobot apparatus 1, transiently holds the sort and the likelihood of theemotion supplied from the user speech recognition unit 103 and/or theuser image recognition unit 104 to send them to the feeling estimationunit 106. It is not necessarily the case that the type and thelikelihood of the emotion are supplied from both the user speechrecognition unit 103 and the user image recognition unit 104. If thetype and the likelihood of the emotion are supplied only from one ofthese units, the short term memory 105 transiently holds the sort andthe likelihood of the supplied emotion. For example, if the user hascalled to the robot apparatus 1 from its back side, the robot apparatus1 is unable to recognize the expressions of the user's face. Thus, onlythe sort and the likelihood of the emotion as detected from the user'svoice are sent from the user speech recognition unit 103.

The feeling estimation unit 106 finds the user's current status e_(c) inthe above-described emotion manifold, based on the sort and thelikelihood of the emotion as detected by the user speech recognitionunit 103 and/or the user image recognition unit 104 and as supplied fromthe short term memory 105, to send the data to the action selection unit108. Based on the emotion parameters and on the instinct parameters ofthe robot apparatus 1, as supplied from the internal status model 102,the feeling estimation unit 106 finds the current status of the robotapparatus 1 in the above-described emotion manifold, to send the data tothe action selection unit 108.

The long term memory 107, responsible for long-term storage for therobot apparatus 1, has stored therein pre-learned plural data triples<e_(i), a, e_(t)> as the aforementioned learning sample l.

The action selection unit 108 corresponds to the behavior model library70 and the behavior switching module 71 shown in FIG. 7. This actionselection unit 108 sets the user's current status e_(c), supplied fromthe feeling estimation unit 106, as the initial status e_(i), and readsout the learning samples <e_(i)′, a, e_(t)′>, approximate to datatriples <e_(i), *, e_(t)>, from the long term memory 107, based on thisinitial status e_(i) and on the desired target status e_(t). The actionselection unit 108 sets the action a in the learning samples l as thetarget behavior t, that is sets t:=a, and selects the target behavior tfrom the behavior models 70 _(t) to 70 _(n) to output the so selectedtarget behavior. In the present embodiment, the current status of therobot apparatus 1, supplied from the feeling estimation unit 106, isused as this target status e_(t). This enables the robot apparatus 1 toswitch the own behavior so as to lead the user's feeling status to afeeling status close to the robot's own emotion.

In this manner, the robot apparatus 1 is able to estimate the user'sfeeling from the user's voice tone or the user's facial expressions and,based on the estimated results, changes over the own action to lead theuser's status of feeling to a status of feeling close to the robot's owncurrent emotion.

The feeling recognition processing in the user speech recognition unit103 and in the user image recognition unit 104, and the processing ofbehavior selection based on the feeling, are hereinafter explained indetail.

(2-3-1) Voice-Based Feeling Recognition

The user speech recognition unit 103 detects a sum of four emotions,namely the joy/pleasure (JOY/PLEASURE), sorrow/sadness/grief(SORROW/SADNESS/GRIEF), angry (ANGRY) and calmness (CALMNESS), based onthe intensity and the fundamental frequency (pitch) of the speechuttered by the user. Meanwhile, the sorts of the emotions detected areof course not limited to the four depicted above. While there are alarge variety of algorithms used for classifying the input speechaccording to respective feelings, as reported by Pierre-yves Oudeyer(Pierre-yves Oudeyer (2001) Algorithm and Features, Proceedings of theHumanoids Conference), a Naive Bayes classification algorithm), is hereused as an example.

Specifically, the user speech recognition unit 103 includes a speechinput unit 111, a characteristic value extraction unit 112, a feelingdetection unit 113, a classifier 114, a feeling speech data base 115 anda result output unit 116, as shown in FIG. 12. It should be noted thatthe classifier 114 includes a joy classifier 114 ₁, a sadness classifier114 ₂, an anger classifier 114 ₃ and a calmness classifier 114 ₄,associated with the aforementioned four emotions, respectively.

Of the speech signals (acoustic signals) from the microphone 23, shownin FIG. 4, the speech uttered by the user is AID converted into digitalspeech signals and input to the speech input unit 111. This speech inputunit sends the digital speech signals to the characteristic valueextraction unit 112.

The characteristic value extraction unit 112 extracts, as characteristicvalues, the intensity and the pitch of the speech from the digitalspeech signals supplied from the speech input unit 111. The intensity ofthe speech is afforded as the signal power, while the pitch of thespeech is afforded as the fundamental frequency of the signals. Thereare a variety of techniques for finding the fundamental frequency, forexample, a technique employing the autocorrelation function of thespeech waveform or a technique employing the spectrum.

The characteristic value extraction unit 112 extracts the intensity andthe pitch of the speech of the digital speech signals at an interval ofan extremely short time interval, such as every 10 ms. In this case, thespeech intensity is found from the signals passed through low-pass andhigh-pass filters. As a result, a sequence of 3-dimensional vectorspertinent to the intensity and the pitch of the speech is found for asole digital speech signal. The characteristic value extraction unit 112finds an average value, a maximum value, a minimum value, a differencebetween the maximum and minimum values, a variance and a median value,for each dimension of the sequence of 3-dimensional vectors. The resultis that 3×6=18 characteristic values are obtained. The characteristicvalue extraction unit 112 sends the 18-dimensional vectors to thefeeling detection unit 113.

The feeling detection unit 113 classifies the 18-dimensional vectors,supplied from the characteristic value extraction unit 112, into thejoy/pleasure (JOY/PLEASURE), sorrow/sadness/grief(SORROW/SADNESS/GRIEF), angry (ANGRY) and calmness (CALMNESS), using therespective classifiers 114 ₁ to 114 ₄ of the classifier 114, inaccordance with the Naive Bayers classification algorithm.

This Naive Bayers classification algorithm is now briefly explained. Theproblem of finding c_j which maximizes the probability of occurrence(P(c_j|a_(—)1, a_(—)2, . . . , a-n) of class c_j for the n-dimensionalinput vector (a_(—)1, a_(—)2, . . . , a-n) is formularized, by theBayes' theorem, into the problem of finding c_j which maximizes thefollowing equation (4):P(a _(—)1, a _(—)2, . . . , a _(—) n(c _(—) j)P(c _(—) j).  (4)Under a hypothesis that there exists independency of the respectivedimensions, the above problem is reduced to a problem of finding theclass c_j which maximizes the following equation (5):P(a _(—)1|c_)P(a _(—)2|c _(—) j) . . . P(a _(—) n|c _(—) j)P(c _(—)j).  (5)

By modeling P(a_i|c_j), using a probability density function afforded byfor example the normal distribution, the occurrence probability of theclass c_j may be found for an input vector P(a_(—)1, a_(—)2, . . . ,a_n).

That is, by calculating the likelihood, using the probability densityfunction associated with respective emotions for each of theabove-mentioned 18-dimensional vectors, it is possible to calculate thelikelihood of the 18-dimensional vectors associated with the respectiveemotions.

This probability density function may be estimated using a feelingspeech database 115 having stored therein a large number of speech dataassociated with respective emotions. For example, if there are 1000utterances of speech data associated with a given feeling, theprobability density function represented by the following equation (6):$\begin{matrix}{{P(x)} = {\frac{1}{\sqrt{2\quad\pi}\sigma}\exp\left\{ {- \frac{\left( {x - \mu} \right)^{2}}{2\quad\sigma^{2}}} \right\}}} & (6)\end{matrix}$may be estimated by extracting the characteristic values (theaforementioned 18-dimensional vectors) for each speech data and byfinding the average value p and the variance a from the data associatedwith respective dimensions of the characteristic value vectors.

The feeling detection uni 113 finds, for the 18-dimensional vectors,supplied from the characteristic value extraction unit 112, thelikelihood of each emotion from the probability density function in thejoy classifier 114 ₁, sadness classifier 114 ₂, anger classifier 114 ₃and in the calmness classifier 114 ₄. The feeling detection uni 113compares this likelihood and sends the sort of the emotion with themaximum likelihood and the likelihood as the detecting result to theresult output unit 116.

The result output unit 116 outputs the sort of the emotion supplied fromthe feeling detection uni 113 and the likelihood, supplied from thefeeling detection uni 113, to the short term memory 105 shown in FIG.11.

2-3-2) Feeling Recognition Based on Facial Expressions

Based on the expressions of the user's facial expressions, the userimage recognition unit 104 detects a sum of six emotions, namely the joy(JOY), sadness (SADNESS), anger (ANGER), surprise (SURPRISE), disgust(DISGUST) and fear (FEAR). Of course, the sorts of the emotions to bedetected are not limited to these six emotions. If the emotion is noneof these six emotions, the emotion of calm/neutral (CALM/NEUTRAL) isdetected.

For describing the facial expressions, the method for expressiondescription proposed by Ekman and Friesen, and which is called thefacial action coding system (FACD), is used in the present embodiment.This FACS classifies the expressions in accordance with minimum units ofthe expression performances, termed action units (AU), which are (a)analytically independent (expression muscles), (b) visualdiscernability, and (c) minimum units of expressional operations, andquantitatively describes the human expressions by the combination of theAU intensity values. However that FACS has inherently been proposed fordescribing and analyzing the expressions in the field of psychology and,as shown in excerpt in FIG. 13, the definitions of the AUs usequalitative expressions.

The expressions demonstrated on occurrence of the above-mentionedrespective six emotions are termed basic six expressions. The relevancebetween these six expressions and the AUs has been clarified by Ekmanand Friesen. Meanwhile, this relevance is basically unaffected by theindividual difference, such as culture, sex or age. That is, if thehuman expressions can be classified or recognized by AU, these can bemapped into the aforementioned six emotions.

Thus, the user image recognition unit 104 exploits the FACS coding, asdata conversion processing, having picture signals and the AU intensityvalues as input and output, respectively, to detect the aforementionedsix emotions, characterized by the respective AUs, from the inputpicture signals.

Referring to FIG. 14, the user image recognition unit 104 includes apicture input unit 121, a face detection unit 122, an alignment unit123, a differential picture generating unit 124, a facial image database 125, a data conversion unit 126, an HMM (hidden Markov model) unit127, and a result output unit 128.

The picture input unit 121 is supplied with picture signals from the CCDcamera 20, shown in FIG. 4, to send the picture signals to the facedetection unit 122.

The face detection unit 122 transforms a frame picture, corresponding topicture signals supplied from the picture input unit 121, into pluralscale pictures having different contraction ratios. For example, theface detection unit 122 transforms the frame picture by sequentiallycontracting the frame picture, by factors of 0.8, into scale pictures offive stages (1.0 times, 0.8 times, 0.64 times, 0.51 times and 0.41 timesthe frame picture size). The face detection unit 122 scans each scalepicture starting from the upper left of the picture and proceedingsequentially to the lower right thereof, by shifting a suitable numberof pixels towards right or towards below, to slice out a rectangulararea of (20×20)=400 pixels. Moreover, the face detection unit 122 checkswhether or not the picture scanned is a face image, by taking matchingto a template picture. Since the rectangular areas found at this timepoint to be a face image, referred to below as a score image, mayfrequently be a mis-verified non-face picture, the face detection unit122 verifies face- non-face discrimination by a support vector machine(SVM).

The above-described technique is disclosed e.g., in a report by B.Sholkoph (B. Sholkoph, C. Burges, A. Smola (1999) Advance in KernelMethods Support Vector Learning, The MIT Press), or in a report by V.Vapnic (V. Vapnic (1999) The Nature of Statistical Learning TheorySecond Edition, Springer).

The aforementioned SVM is a learning machine employing a lineardiscriminator (Perceptron) as a discriminating function and which may beexpanded to a non-linear domain by employing a kernel function. Sincethe learning of the discrimination function is carried out such as tomaximize the margin of class-to-class separation, and the solution isobtained by solution using the quadratic mathematical programmingmethod, arrival at a global solution may be theoretically guaranteed.The following are the details of the processing:

In order to remove a non-face background portion from the aforementionedscore picture, 360 pixels are extracted from the score picture having20×20=400 pixels, using a mask corresponding to the picture the fourcorners of which have been cut off. Then, supposing that the luminosityof an object being imaged is changed by illumination, histogramsmoothing or contrast normalization is carried out. An identificationfunction used for face detection then is learned. Using each presetnumber of the face data and the non-face data, as the learning data, aprovisional identification function is first obtained. Theidentification function, obtained provisionally, then is tested on avariety of images on the data base to detect a face. Those images whichwere successfully detected to be face images are added to the learningdata as face data, while those which resulted in failure are added tothe learning data as non-face data and subjected to re-learning.

The face detection unit 122 thus verifies whether or not there are facedata in the score picture. If the face data exist, the score picture issent as face image to the alignment unit 123.

The alignment unit 123 executes alignment processing on a face imagesupplied from the face detection unit 122. That is, for obtaining acorrect difference between two face images in the differential picturegenerating unit 124 to be described later, the face position needs to befound accurately. Moreover, since the positions of the eyes and the nosein a face differ from person to person, these positions need to benormalized. Consequently, the alignment unit 123 extracts characteristicpoints, such as eye ends, nostrils or lip ends, of a given face image toidentify eye and nose positions. The positions of the eyes and the noseas detected are rotated and expanded/contracted (morphed) by affinetransformation to execute alignment processing. The alignment unit 123sends the face image following alignment to the differential picturegenerating unit 124.

The differential picture generating unit 124 calculates the differencebetween the post-alignment face image, supplied from the alignment unit123, and the post-alignment expressionless (neutral) face image storedin the facial image data base 125 to generate a differential image. Thedifferential picture generating unit 124 sends the generateddifferential image to the data conversion unit 126.

The data conversion unit 126 performs vector transformation on the360-pixel score picture, using a Gabor filter, to transform a set of thevectors produced further into a sole characteristic vector.

It has already been known that there exist the cells of the human visualsystem which exhibit selectivity to a certain specified orientation.These cells are made up by those reacting to a vertical line and thosereacting to a horizontal line. The aforementioned Gabor filter is aspatial filter composed of plural filters exhibiting this orientationselectivity.

With filtering at a lower frequency, it may be advisable to decrease thevector dimensions by downsampling because it is redundant to hold thepost-filtering images as vectors in their entirety. In this case, thedownsampled plural vector sets are arrayed to generate a solecharacteristic vector.

The data conversion unit 126 vector-quantizes the feature vectors,obtained on Gabor filtering, to send the resulting vector-quantizedcharacteristic vectors to the HMM unit 127.

The HMM unit 127 classifies the vector-quantized characteristic vectors,supplied from the data conversion unit 126, into one of the joy (JOY),sadness (SADNESS), anger (ANGER), surprise (SURPRISE), disgust (DISGUST)and fear (FEAR). Meanwhile, the HMM unit 127 has been pre-trained withface images expressing respective emotions and finds the likelihood ofeach emotion for the supplied characteristic vectors. Meanwhile, therespective emotions and the AUs may, for example, be correlated asfollows:

-   joy (JOY) . . . AU6+12-   sadness (SADNESS) . . . AU1 and AU 1+4-   anger (ANGER) . . . AU4-   surprise (SURPRISE) . . . AU1+2-   disgust (DISGUST) . . . AU2+4-   fear (FEAR) . . . AU2+2+4

The HMM unit 127 compares the likelihoods of respective emotionsobtained to one another to send the type of the emotion having themaximum likelihood and the likelihood as detected results to the resultoutput unit 128.

The result output unit 128 outputs the sort of the emotion and thelikelihood thereof, supplied from the HMM unit 127, to the short termmemory 105 shown in FIG. 11.

In the foregoing, an input to the HMM unit 127 is the characteristicvector obtained on Gabor filtering and which subsequently is processedwith vector quantization. This is merely illustrative such that it isalso possible to use results of tracking of characteristic points andthe motion vector (optical flow) of various face sites as disclosed forexample in a report by James J. Lien (Janes J. Lien et al. (1998)Automated Facial Expression Recognition Based on FACS Action Units,proceedings of FG'98, April 14-16).

(2-3-3) Behavior Selection Based on Feeling

The robot apparatus 1 has a total of six emotions, namely joy (JOY),sadness (SADNESS), anger (ANGER), surprise (SURPRISE), disgust (DISGUST)and fear (FEAR), as described above. These respective emotions areassociated with spatial regions in a 3-dimensional topological spacehaving three axes of pleasantness (PLEASANTNESS), arousal (AROUSAL) andcertainty (CERTAINTY), as shown for example in FIG. 15.

Of these, the [pleasantness] is a parameter indicating the degree ofsatisfaction of the instinct of the robot apparatus 1. On the otherhand, the [arousal] and [certainty] are parameters indicating the degreeof awakened or sleeping status of the instinct of the robot apparatus 1and the degree of certainty or confidence with which the robot apparatus1 is able to recognize the subject to which the robot apparatus 1 iscurrently directing its attention.

The feeling estimation unit 106, shown in FIG. 11, first maps the user'scurrent feeling on a point in the topological space, based on the sortand the likelihood of the emotion detected by the user speechrecognition unit 103 and/or the user image recognition unit 104 andsupplied from the short term memory 105. Specifically, when the emotiondetected is the joy (JOY), with its likelihood being 0.8, the feelingestimation unit 106 multiplies a vectorj, proceeding from a point oforigin of the topological space towards the center of the spatial areacorresponding to the detected emotion ofjoy, with the likelihood of 0.8,as shown in FIG. 16. The coordinate J (p, a, c) indicated by the soproduced vector is provisionally set as a coordinate specifying theuser's current feeling.

Meanwhile, if the likelihood of the emotion detected by the user speechrecognition unit 103 differs from the likelihood of the emotion detectedby the user image recognition unit 104, the feeling estimation unit 106is able to map the current feeling using an average value of the twolikelihoods. If the emotion detected by the user speech recognition unit103 differs from the emotion detected by the user image recognition unit104, the feeling estimation unit 106 is able to map the current feelinginto the topological space using only a preset one of the two emotions.

The feeling estimation unit 106 projects a coordinate in the topologicalspace, representing the user's current feeling, on a topological planehaving two axes of, for example, [pleasantness] and [arousal], and setsthe resulting coordinate as the user's current status e_(c). Similarly,the feeling estimation unit 106 projects a point in the topologicalspace representing the robot apparatus' current status (feeling) on atopological plane having two axes of [pleasantness] and [arousal] andsets the resulting coordinate as the user's target status e_(t).

The action selection unit 108 sets the user's current status e_(c), assupplied from the feeling estimation unit 106, as the initial statuse_(i), and presupposes an action * as being an ideal target action whichis to lead the user from this initial status e_(i) to the target statuse_(i). That is, the initial status e_(i), afforded as a coordinate onthe topological plane having two axes of [pleasantness] and [arousal],is transformed by the action * into a target status e_(i) afforded asanother coordinate on the topological plane, as shown in FIG. 17. Theaction selection unit 108 reads out the learning sample l=<e_(i)′, a,e_(t)′> approximate to the data triple <e_(i), *, e_(t)> from the longterm memory 107. The action selection unit 108 sets the action a in thelearning sample l as a target action t, that is sets t:=a, and selectsthis target action t from the behavior models 70 ₁ to 70 _(n) to outputthe selected behavior model.

In this manner, the robot apparatus 1 changes over its own behavior inorder to lead the user's status of feeling to a feeling status close tothe robot apparatus' own current emotion.

The method for controlling the behavior of the robot apparatus 1described above is now explained with reference to the flowchart of FIG.18.

First, in a step S20, the robot apparatus 1 is initialized on startup ofthe action control operations. In a next step S21, the initial statuse_(i), that is the user's current feeling, is detected as the currentstatus e_(c):e_(i)=e_(c).

In a next step S22, the current feeling of the robot apparatus 1 is setas the target status e_(t) to which the user should be led.

In a step S23, it is checked whether or not there is any learning sampleclose to <e_(i), *, e_(t)> among the learning samples <e_(i)′, a,e_(t)′> stored in the long term memory 107. Specifically, a distanceDist represented by the following equation (7): $\begin{matrix}\begin{matrix}{{Dist} = {{comp}\left( {{\text{<}{ei}},{*{,{{ei}\text{>}},{\text{<}{ei}^{\prime}},a,{{et}^{\prime}\text{>}}}}} \right)}} \\{= {{\alpha{{{ei} - {ei}^{\prime}}}} + {\beta{{{et} - {et}^{\prime}}}}}}\end{matrix} & (7)\end{matrix}$where α and β are preset constants, is compared to a threshold value Thto check whether or not the distance Dist is smaller than the thresholdvalue Th.

If the distance Dist is smaller than the threshold value Th, it isassumedthat the distance Dist isclose to the threshold value Th. Thus,the learning sample l is read in step S24 from the long term memory 107and the action to be derived is set as a target action t:=a. On theother hand, if the distance Dis, is not smaller than the threshold valueTh, another learning sample l=<e_(i)′, a′, e_(t)′> is read out from thelong term memory 107. In a step S26, the action to be derived is set asthe target behavior t:=a′.

In a next step S27, the target action t derived is carried out.

In a step S28, the final status of the user e_(t), as being the resultof the target action t executed, is detected. In a next step S29, thedata triple <e_(i), t, e_(t)> is transiently stored.

In a next step S30, the learning sample l is corrected or a new datatriple <e_(i), t, e_(t)> is stored in the long term memory 107. Forexample, the distance Dist₁ between the data triple <e_(i), *, e_(t)>and the data triple <e_(i)′, a, e_(t)′> is compared to the distanceDist₂ between the data triple <e_(i), *, e_(t)> and the data triple<e_(i)′, a, e_(t)′>. If the distance Dist₁ is smaller than the distanceDist₂, the data triple <e_(i)′, a, e_(t)′> in the learning sample l isreplaced and corrected by a data triple <e_(i)″, a, c_(t)″> defined bythe following equation (8):ei″=ei′+k·eiet″=et′+k·et  (8)where k is a preset constant such that |k|<1. If conversely the distanceDist₂ is not smaller than the distance Dist₁, the data triple <e_(i), t,e_(t)> is added to the learning sample l and stored in the long termmemory 107.

The steps as from step S21 is repeated until the execution is terminatedby an external control device or by the user.

With the above-described robot apparatus 1, it is possible, by applyingthe method and the system for controlling the man-machine interfaceunit, described above, to estimate the user's feeling from the tone ofthe user's speech or the user's facial expressions, and to lead thestatus of the user's feeling to a feeling status close to the robotapparatus 's own status of feeling, based on the estimated results.

The present invention is not limited to the above-described embodiments,but may be changed variably without departing from its scope.

For example, although the above embodiment is directed to a case wherethe current status of the feeling of the robot apparatus 1 is the targetfeeling e, to which the user should be led, this is for the sake ofillustration only, such that any desired feeling status can be set astarget status e_(t).

Although the status of the user or the robot apparatus 1 is representedin the above embodiment as being a point on the topological plane, thisis merely illustrative, such that it may also be represented as being apoint on the topological plane, shown for example in FIG. 15.

INDUSTRIAL APPLICABILITY

With the above-described method and system for controlling theman-machine interface unit, according to the present invention, theaction of the robot apparatus may be set so as to lead a user from anoptional current status to a desired target status by pre-collecting adata triple comprised of an initial status (feeling) prior to executionof an action, an final status after execution of the action (feeling)and the action taken place, as learning sample. In addition, by applyingthe method and system for controlling the man-machine interface unit toan autonomous robot apparatus, at least having a feeling model and whichis able to change the action expressed responsive to the feeling, it ispossible for the robot apparatus to change the action so as to lead theuser's feeling status to a feeling status close to the robot apparatus'own current emotion, based on the estimated results of the user'sfeeling, thus improving entertainment performance of the robotapparatus.

1. A method for controlling a man-machine-interface unit, wherein withrespect to at least one user an initial status (e_(i)) before and afinal status (e_(f)) after an action (a) taken place are essentiallydetected on said man-machine-interface, and wherein data triplesessentially representing said initial status (e_(i)), said final status(e_(f)) and said action (a) taken place are collected as learningsamples (1), and wherein said learning samples (1) are used to at leastderive and initialize a target action to be carried out so as to carryin each case said user from a given current status (e_(c)) to a givendesired target status (e_(t)) on said man-machine-interface.
 2. A methodaccording to claim 1 wherein in each case current and initial statusdata (CSD, ISD) are received and generated essentially corresponding toand representing said current status (e_(c)) and the initial status(e_(i)) of said user, respectively.
 3. A method according to claim 1wherein in each case final and target status data (FSD, TSD) arereceived and generated essentially corresponding to and representingsaid final status (ef) and said target status (et) of said user,respectively.
 4. A method according to claim 1 wherein action data (AD)and target action data (TAD) are received and generated essentiallycorresponding to and representing said action (a) taken place and saidtarget action (t) derived.
 5. A method according to claim 1 wherein saidcurrent, initial, final and target statuses (ec, ei, ef, et) and inparticular the corresponding and/or representing data (CSD, ISD, FSD,TSD) thereof are in each case represented by and modeled in an embeddingtopological manifold (EM, DM), in particular of two dimensions.
 6. Amethod according to claim 5 wherein separate manifolds are derivedand/or used for distinct users and/or for classes of users and/orwherein common or global manifolds for all users are derived and/orused.
 7. A method according to claim 1 wherein said action (a) takenplace, said target action (t) derived said representing andcorresponding data (AD, TAD) thereof are in each case represented by andmodeled in an embedding topological manifold (AM, ADM).
 8. A methodaccording to claim 1 wherein emotional statuses are used said current,initial, final, target statuses (ec, ei, ef, et).
 9. A method accordingto claim 1 wherein for each user based on the set (L) of learningsamples (1) a transmotion mapping (T)E⊕A→E<ei, a>E⊕A→ef: =T(ei, a)=Ta(ei) E is derived and wherein E denotes anemotion manifold and is one of the topological manifolds (EM, EDM) forthe user statuses and for the status data thereof, A denotes an actionmanifold and is one the topological manifolds (AM, ADM) for the actionand for the action data and ei, ef and a denote respective elements ofthe manifolds E and A.
 10. A method according to claim 1 wherein forgiven desired initial and target statuses (ei, et) for a given user anaction (a) from a learning sample (1) with a given current status (ec),final status (ef) and an action (a) taken place already sampled ischosen as a target action (t) for which the initial and current statuses(ei, ec) and the final or the target statuses (ef, et) and the data(ISD, CSD, FSD, TSD) thereof are in a next topological neighborhood. 11.A method according to claim 10 wherein a distance measure, a metric,continuity, direction, vector properties of said statuses (e_(i), e_(e),e_(T), e_(t)) and actions (a, t) are involved to derive and describesaid neighborhood.
 12. A method according to claim 1 wherein a sequenceof actions (a) to be carried out is derived and initialized to beexecuted as said target action (t) so as to lead a given user to atarget status (et) to fit best to a given desired final status (ef) forsaid user.
 13. A method according to claim 1 wherein a modification isincluded into and performed on a derived target action (t) and into thedata (TAD) thereof.
 14. A method according to claim 13 wherein saidmodification is introduced randomly, by means of a probabilisticmeasure, on the basis of topological, distance, metric, continuity,direction, vector properties.
 15. A method according to claim 1 whereinobtained learning samples (1)—in particular to derive said topologicalmanifolds (EM, EDM, AM, ADM)—are generalized.
 16. A method according toclaim 15 wherein the step of generalizing said learning samples (1) atleast comprises a step of averaging and weighting learning samples (1)from different users, in particular to derive common and/or globaltopological manifolds, wherein in particular said different users areemotionally clustered depending on their typical status/and or behavior.17. A method according to claim 15 wherein the step of generalizing saidlearning samples (1) at least comprises at least a step of reducing thedimension of the action manifold (AN, ADM), in particular by means of aprojection.
 18. A method according to claim 5 wherein the step ofgeneralizing said learning samples (1) comprises at least a step ofenlarging and reducing the extension of neighborhood of a given learningsample (1) within which all triples of initial statuses (ei), finalstatuses (ef) and target actions (t) are classified as being equivalentto said given learning sample (1), in particular until the point where afinal status (ef) and a target action (t) are valid for all initialstatuses (ei) and until the point where the final status (ef) isirrelevant and only a direction of a change of the statuses isconsidered.
 19. A system for controlling a man-machine-interface unitwhich is capable of performing and realizing the method for controllinga man-machine-interface unit according to claim 1 and the steps thereof.20. A computer program product comprising computer program means adaptedto perform and realize the method for controlling aman-machine-interface unit according to claim 1 and the steps thereof,when it is executed on a computer, a digital signal processing means.21. A robot apparatus acting responsive to inner and outer statusescomprising: status detection means for detecting the status of a user ofthe robot apparatus; target status setting means for setting a desiredtarget status of the user; action determining means for determining anaction to be carried out to lead the user's status detected by saidstatus detection means to said target status; and controlling means forcarrying out the determined action.
 22. The robot apparatus according loclaim 21 further comprising: learning means; wherein the action to becarried out to lead the user's status as detected by said statusdetection means to said target status is updated by said learning means.23. The robot apparatus according to claim 22 wherein said learningmeans collects a data triple representing the initial status before anda final status after an action taken place, and the action taken place,as learning sample, in connection with said user; and wherein saidaction determining means decides on said action to be carried out basedon said learning sample.
 24. The robot apparatus according to claim 21wherein said user's status and the target status are the status of theuser's feeling.
 25. The robot apparatus according to claim 24 whereinsaid target status setting means uses the feeling status of said robotapparatus as said target status.
 26. The robot apparatus according toclaim 24 further comprising: speech recognition means and picturerecognition means; said status detection means detecting the feelingstatus as said user's status from the speech and a face image of saiduser.
 27. The robot apparatus according to claim 24 wherein said feelingstatus may be represented as a coordinate in topological space of amultiple dimensions; and wherein the user's status and the target statusare given as coordinates in said topological space.
 28. The robotapparatus according to claim 24 wherein said feeling status may berepresented as a coordinate in a multi-dimensional topological space;and wherein the user's status and the target status are given ascoordinates obtained by projecting the coordinate in said topologicalspace on a topological plane comprising two of said multiple dimensions.29. A method for controlling the behavior of a robot apparatus actingautonmously responsive to inner and outer statuses, comprising: a statusdetection step of detecting the status of a user of the robot apparatus;a target status setting step of setting a desired target status of theuser; an action determining step of determining an action to be carriedout to lead the user's status as detected by said status detection stepto said target status; and a controlling step of carrying out the actiondetermined.
 30. The method according to claim 29 further comprising: alearning step; wherein the action to be carried out to lead the user'sstatus as detected by said status detection step to said target statusis updated by said learning step.
 31. The method according to claim 30wherein in said learning step, a data triple representing the initialstatus before and a final status after an action taken place, and theaction taken place, as learning sample, in connection with said user, iscollected; and wherein said action determining step decides on saidaction to be carried out based on said learning sample.
 32. The methodaccording to claim 29 wherein the user's status and the target statusare the user's feeling status.
 33. The method according to claim 32wherein, in said target status setting step, the feeling status of therobot apparatus is used as said target status.
 34. The method accordingto claim 32 further comprising: a speech recognition step and an imagerecognition step; said status detection step detects the feeling status,as said user's status, from the speech and the face image of the user.35. The method according to claim 32 wherein said feeling status may berepresented as a coordinate in topological space of a multipledimensions; and wherein the user's status and the target status aregiven as coordinates in said topological space.
 36. The method accordingto claim 32 wherein said feeling status may be represented as acoordinate in a multi-dimensional topological space; and wherein theuser's status and the target status are given as coordinates obtained byprojecting the coordinate in said topological space on a topologicalplane comprising two of said multiple dimensions.